Targets, fixtures, and workflows for calibrating an endoscopic camera

ABSTRACT

The present disclosure relates to calibration assemblies and methods for use with an imaging system, such as an endoscopic imaging system. A calibration assembly includes: an interface for constraining engagement with an endoscopic imaging system; a target coupled with the interface so as to be within the field of view of the imaging system, the target including multiple of markers having calibration features that include identification features; and a processor configured to identify from first and second images obtained at first and second relative spatial arrangements between the imaging system and the target, respectively, at least some of the markers from the identification features, and using the identified markers and calibration feature positions within the images to generate calibration data.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a division of U.S. application Ser. No. 14/828,781(filed Aug. 18, 2015), which is a division of U.S. application Ser. No.13/535,011 (filed Jun. 27, 2012), now U.S. Pat. No. 9,134,150, which isa continuation of U.S. application Ser. No. 12/415,377 (filed Mar. 31,2009), now U.S. Pat. No. 8,223,193, each of which is incorporated hereinby reference.

BACKGROUND

Minimally invasive surgical techniques are aimed at reducing the amountof extraneous tissue that is damaged during diagnostic or surgicalprocedures, thereby reducing patient recovery time, discomfort, anddeleterious side effects. As a consequence, the average length of ahospital stay for standard surgery may be shortened significantly usingminimally invasive surgical techniques. Also, patient recovery time,patient discomfort, surgical side effects, and time away from work mayalso be reduced with minimally invasive surgery.

A common form of minimally invasive surgery is endoscopy, and a commonform of endoscopy is laparoscopy, which is minimally invasive inspectionand surgery inside the abdominal cavity. In standard laparoscopicsurgery, a patient's abdomen is insufflated with gas, and cannulasleeves are passed through small (approximately ½ inch or less)incisions to provide entry ports for laparoscopic instruments.Laparoscopic surgical instruments generally include a laparoscope or anendoscope for viewing the surgical field.

An endoscope can be calibrated prior to use. Calibration is the processof determining intrinsic and extrinsic parameters for an imaging deviceby projecting three-dimensional (3-D) points into an image. Intrinsicparameters involve the internal geometric and optical characteristics ofthe imaging device, such as focal lengths in x and y, principal point inx and y, skew and pixel aspect ratio, and distortions (often quantifiedby a few parameters describing the distortions such as radial andtangential distortions). Intrinsic parameters can be used to compensatefor imaging errors, such as optical aberrations of the imaging device.Extrinsic parameters involve the 3-D position of the camera referencecoordinate system relative to a certain world coordinate system (i.e.,six degree of freedom pose). In general, calibration is essential formany advanced imaging systems, such as advanced computer vision, 3-Daugmented reality, 3-D visualization applications, advanced userinterfaces, and image-guided surgery.

A stereoscopic imaging device, such as a stereo endoscope, is typicallyaligned at some point prior to use. The alignment process involvesadjusting the left and right stereo images horizontally and verticallyso as to have zero horizontal and vertical disparity at a certaindistance. Without alignment, a viewer's eyes cannot properly fuse theleft and right images (especially if the vertical disparity is large).Exemplary alignment methods and systems are described in commonly ownedU.S. Pat. No. 7,277,120 (filed Mar. 7, 2004), which is herebyincorporated by reference. Calibration parameters for the two imagingpaths of a stereo imaging device can provide parameters (horizontal andvertical offsets) of the alignment process.

Typical calibration methods involve imaging a calibration target. Acalibration target typically has multiple features having known targetrelative coordinates. An image of the calibration target is processed soas to determine a collection of image coordinates associated with atleast some of the target features. Known calibration methods can be usedto process the collection of associated coordinates so as to generatecalibration parameters, both extrinsic and intrinsic. (For exemplarymethods, see Z. Zhang, “A flexible new technique for cameracalibration,” IEEE trans. Pattern Analysis and Machine Intelligence,2000, volume 22, number 11, pages 1330-1334; and Janne Heikkila and OlliSilven, “A Four-step Camera Calibration Procedure with Implicit ImageCorrection,” available at url<www.vision.caltech.edu/bouguetycalib_doc/papers/heikkila97.pdf>, whichare both hereby incorporated by reference.) Another method isimplemented in a Matlab toolbox by Jean-Yves Bouguet (available at url<www.vision.caltech.edu/bouguetycalib_doc/index.html>), which is aslightly modified version of the method described in the above-listedZhang reference.

Calibration targets can be 3-D, two-dimensional (2-D), orone-dimensional (1-D). A 2-D target and related method(s) have a goodbalance of accuracy and convenience and are preferred in manyapplications. Calibration using planar targets requires multiple imagesof the target at different orientations so that the features beingimaged have coordinates in three dimensions in any possible referencecoordinate system, which is typically required by the matrix operationsused to process the collection of associated coordinates. The exactposes of the target do not need to be known, since they can be estimatedin the calibration process.

Existing methods used to extract obtain and process images ofcalibration targets suffer from a number of problems. For example, onecalibration method involves imaging a checkerboard target pattern. Thecheckerboard pattern target must be properly positioned/orientedrelative to the imaging device for multiple imaging directions. Butproperly placing the pattern as required by a calibration algorithm isnot intuitive, and placement may therefore be difficult to guarantee. Itcan be especially difficult for non-technical persons to followinstructions directed to obtaining sufficiently different imagingdirections. Additionally, since human hands are not very steady, holdingthe camera or target freehand typically induces motion blur. Somemethods require manually designating corners of the pattern in theresulting images, such as the Matlab camera calibration tool box (seeprevious reference). As another example, the OpenCV computer visionlibrary needs to have the number of grids of the pattern and requiresthat the full pattern be visible in an image.

There are some calibration methods that do not require manualdesignation. Some attach an attached optical tracking target to thecalibration target to directly determine the 3-D information of thecalibration target features (see Ramin Shahidi, Michael R. Bax, CalvinR. Maurer, Jr., Jeremy A. Johnson, Eric P. Wilkinson, Bai Wang, Jay B.West, Martin J. Citardi, Kim H. Wanwaring, and Rasool Khadem,“Implementation, Calibration and Accuracy Testing of an Image-EnhancedEndoscopy System,” In IEEE Transactions on Medical Imaging, Vol. 21, No.12, December 2002). Some add a few special features in the middle of thepattern that can be used to align the pattern with the image (seeChristian Wengert, Mireille Reeff, Philippe C. Cattin, and GaborSzekely, “Fully Automatic Endoscope Calibration for Intraoperative Use,”In Bildverarbeitung fur die Medizin Hamburg, 2006). However, thisrequires that the special pattern to be visible in an image, whicheliminates the potential use of non-overlapping images of the target. Assuch, further improvements in calibration target design remaindesirable, particularly target features that can be readily associatedwith their resulting images. More recently, some use self-identifyingpatterns for camera calibration (see Mark Fiala and Chang Shu,“Self-identifying patterns for plane-based camera calibration,” InMachine Vision and Applications (2008) 19:209-216). However, it does notprovide a physical device/feature to interface with the imaging deviceto ensure that sufficient orientation variations have been captured andease of use by non-technical users.

An endoscopic imaging system may also have its color balance (such aswhite balance) adjusted. In image processing, color balance involves theadjustment of the intensities of colors, typically the red, green, andblue primary colors. An important goal of this adjustment is to renderspecific colors correctly, particularly neutral colors. There areseveral aspects of image acquisition and display that result in a needfor color balancing, including: that typical imaging device sensors donot match the sensors in the human eye, that the properties of thedisplay medium impact the rendering of the color, and that the ambientconditions for the acquisition of the image may differ from the displayviewing conditions. Color balance adjustment to keep neutral colors,such as white, neutral is sometimes called gray balance, neutralbalance, or white balance, and this adjustment is a particularlyimportant, if not dominant, element of color balancing.

It may also be advantageous to subject an endoscopic imaging system todiagnostic testing from time to time. A typical endoscopic imagingsystem includes a variety of components, such as imaging sensors, lensassemblies, etc., that may functionally degrade or fail over time. Wherefunctional degradation that does not rise to an intolerable level hasoccurred, an endoscopic imaging system may continue to be used due to alack of knowledge on the part of the user that any functionaldegradation has occurred. Such latent functional degradation may havesignificant detrimental consequences in a critical image-guidedprocedure, such as many minimally invasive surgeries.

While imaging device calibration, alignment, color balance, anddiagnostic testing may be performed by using existing methods anddevices, improved methods and assemblies for performing these tasks in amore convenient and efficient manner remain of interest. For example,methods and assemblies that can be conveniently used to perform thesetasks all at once prior to a surgery would be of particular interest.

BRIEF SUMMARY

In accordance with various aspects, improved assemblies and methods areprovided for generating calibration data, color balance data, anddiagnostic data for an imaging device. Such assemblies and methods canbe particularly advantageous when used to calibrate, adjust the colorbalance on, or run a diagnostic test on an endoscope prior to use. Theprovided assemblies and methods can be used, for example, to reduce theamount of time and labor required to calibrate an endoscope prior touse. The provided assemblies and methods may be less prone to errors inthe form of accidentally missed steps, and they may result in improvedimaging due to image system calibration and color balancing, as well asby avoiding the use of a functionally degraded endoscope.

Thus, the following presents a simplified summary of some embodiments ofthe invention in order to provide a basic understanding of theinvention. This summary is not an extensive overview of the invention.It is not intended to identify key/critical elements of the invention orto delineate the scope of the invention. Its sole purpose is to presentsome aspects and embodiments of the invention in a simplified form as aprelude to the more detailed description that is presented later.

In accordance with an embodiment, a calibration assembly for use with anendoscopic imaging system having a field of view is provided. Thecalibration assembly includes an interface configured for constrainingengagement with the endoscopic imaging system, a target coupled with theinterface so as to be within the field of view, and a processor coupledwith the imaging system. The calibration assembly is reconfigurable froma first relative spatial arrangement between the imaging system and thetarget to a second relative spatial arrangement between the imagingsystem and the target. The target includes multiple markers havingcalibration features that include identification features. The processoris configured to identify, from first and second images obtained at thefirst and second relative spatial arrangements, respectively, at leastsome of the markers from the identification features. The processor isconfigured to generate calibration data by using the identified markersand calibration feature positions within the images.

A calibration assembly can involve a number of options. For example, acalibration assembly can include a tangible medium that includesmachine-readable instructions executable by the processor for processingthe images. The interface can include a receptacle shaped to interfacewith a cylindrical portion of the imaging system. The target can bereconfigurable from the first relative spatial arrangement to the secondrelative spatial arrangement by reorienting the imaging system relativeto the interface, and/or by reorienting the target relative to theinterface. The calibration assembly can include a first portion having afixed spatial arrangement relative to the interface and a second portionhaving a fixed spatial arrangement relative to the target, wherein thefirst and the second portion are rotationally coupled, or wherein thefirst and second portions are coupled so as to provide a plurality ofdifferent target normal directions and/or distances with respect to theimaging system. The features of one or more markers can be arranged in atwo-dimensional pattern, and/or in a one-dimensional pattern. Theprocessor can be configured to generate relative modulation transferfunction (MTF) values for an image of the target. The target can includea straight edge feature separating dark and bright regions of the targetand the processor can be configured to generate MTF values by processingan image region of interest that includes the edge feature. Theprocessor can be configured to generate a color balance adjustment forthe imaging system by processing a region of interest of an image of thetarget, and the color balance adjustment can include a white balanceadjustment. The target can include a white background and non-whitecalibration features. The imaging system can include a stereoscopicendoscope.

In accordance with another embodiment, a calibration assembly for usewith a stereoscopic endoscope imaging system having a field of view isprovided. The calibration assembly includes an interface configured forconstraining engagement with the endoscopic imaging system, a targetcoupled with the interface so as to be within the field of view, and aprocessor coupled with the imaging system. The calibration assembly isconfigurable to a relative spatial arrangement between the imagingsystem and the target. The processor is configured to determine one ormore metrics for the imaging system. The one or more metrics canindicate whether a predetermined Focus Function focal positiondifference is exceeded, whether a predetermined Focus Function peakvalue difference is exceeded, whether a predetermined alignment shiftvalue is exceeded, or whether an illumination level falls below apredetermined illumination level.

In accordance with another embodiment, a calibration target for use incalibrating an imaging system is provided. The calibration target caninclude multiple markers with each marker including multiple localizerfeatures and multiple identification features. The localizer featureshave known relative positions on the target and can be used to determinean orientation for each marker. The identification features can be usedto determine an identification for each marker to establishcorrespondence between image features and target features.

In accordance with another embodiment, a method for calibrating anendoscopic imaging system having a field of view is provided. The methodinvolves using a calibration fixture having an interface forconstraining engagement with the endoscopic imaging system and a targetcoupled with the interface so as to be within the field of view andreconfigurable between multiple spatial arrangements between the imagingsystem and the target. The method includes: using the calibrationfixture to establish a first relative spatial arrangement between thetarget and the imaging system; imaging the target with the imagingsystem from the first relative spatial arrangement, the target includingmultiple markers that include calibration features that includeidentification features, the calibration features having known targetrelative locations; using the calibration fixture to establish a secondrelative spatial arrangement between the target and the imaging system;imaging the target with the imaging system from the second relativespatial arrangement; processing the images to generate data from thecalibration features by identifying at least some of the markers usingthe identification features; and using the generated data to calibratethe endoscopic imaging system.

A method for calibrating an endoscopic imaging system can involve anumber of options. For example, a method can include processing an imageto generate relative MTF values. A method can include processing aregion of interest of one of the images to determine MTF values from astraight edge feature that separates dark and bright regions. A methodcan include processing a region-of interest of one of the images todetermine one or more color balance parameters for the imaging systemand the one or more color balance parameters can include one or morewhite balance parameters. A method can be used with an imaging systemthat includes a stereoscopic endoscope. A method can include determiningone or more metrics for the imaging system. A metric can indicatewhether a predetermined Focus Function focal position difference isexceeded, whether a predetermined Focus Function peak value differenceis exceeded, whether a predetermined alignment shift value is exceeded,or whether an illumination level falls below a predeterminedillumination level. The target imaged can include calibration featuresthat include multiple localizer features that can be used to determinean orientation for each marker.

In accordance with another embodiment, a method for calibrating anendoscopic imaging system having a field of view is provided. The methodinvolves using a calibration fixture having an interface forconstraining engagement with the endoscopic imaging system and a targetcoupled with the interface so as to be within the field of view andreconfigurable between multiple relative spatial arrangements betweenthe imaging system and the target. The method includes: using thecalibration fixture to establish a first relative spatial arrangementbetween the target and the imaging system; imaging the target with theimaging system from the first relative spatial arrangement, the targetincluding multiple features defining multiple local patterns, thefeatures having known target relative locations; using the calibrationfixture to establish a second relative spatial arrangement between thetarget and the imaging system; imaging the target with the imagingsystem from the second relative spatial arrangement; processing theimages to generate data that includes correspondences between imagefeatures and target features by: detecting target features; and locatinglocal patterns by processing the detected features. The processing ofthe images can include identifying unique correspondences between imagefeatures and target features.

For a fuller understanding of the nature and advantages of the presentinvention, reference should be made to the ensuing detailed descriptionand the accompanying drawings. Other aspects, objects and advantages ofthe invention will be apparent from the drawings and the detaileddescription that follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic 3-D view of a stereo imaging system.

FIG. 2 diagrammatically illustrates a calibration assembly and animaging system.

FIGS. 3A, 3B, 4A, 4B, 5, 6A, 6B, 6C, 7A, and 7B are cross-sectionalviews of calibration fixtures.

FIG. 8 diagrammatically illustrates a 2-D marker having localizerfeatures and identification features.

FIGS. 9A and 9B show respective images from two different viewingdirections of a calibration target having multiple 2-D markers and aslanted-edge modulation transfer function (MTF) feature.

FIG. 10 is a flow diagram for a method for determining MTF value(s),color/white balance parameters, calibration parameters, and alignmentparameters.

FIG. 11 is a flow diagram for a method for associating target 2-D markercoordinates with associated image coordinates.

FIGS. 12A through 12E illustrate steps for processing an image so as toidentify a calibration target marker.

FIG. 13A, illustrates exemplary locations within an image of acalibration target of regions of interest that can be processed todetermine color/white-balance parameters and MTF values.

FIGS. 13B and 13C illustrate a non-color balanced region-of interest ofthe image of FIG. 13A and a color balance control.

FIGS. 14A and 14B illustrate calibration targets.

FIG. 15 is a flow diagram for a method that can be used for processingan image of the calibration target of FIG. 14B.

FIG. 16A illustrates a typical normalized Focus Function for a range ofimaging system focal positions.

FIG. 16B illustrates the first derivative of the Focus Function of FIG.16A.

FIG. 17 illustrates a Focus Function algorithm.

FIG. 18 illustrates a number of Focus Function derived metrics that canbe used to check the health of an imaging system.

FIG. 19 illustrates a typical MTF curve for an imaging system.

FIG. 20A shows an tissue image.

FIG. 20B shows a version of the image of FIG. 20A that has been blurredusing a 4×4 Gaussian kernel.

FIGS. 20C and 20D each show the 2-D fast Fourier transform for theimages of FIGS. 20A and 20B, respectively.

FIG. 21 shows relative MTF curves for the images of FIGS. 20A and 20B,respectively.

FIG. 22A shows the tissue image of FIG. 20A.

FIG. 22B shows a version of the image of FIG. 22A that has been blurredusing a 3×3 Gaussian kernel.

FIGS. 22C and 22D each show the 2-D fast Fourier transform for theimages of FIGS. 22A and 22B, respectively.

FIG. 23 shows relative MTF curves for the images of FIGS. 22A and 22B,respectively.

FIG. 24 is a diagrammatic view of two coordinate systems.

FIG. 25A is a plan view of a target pattern, FIG. 25B is a plan view ofa pre-warped target pattern, and FIGS. 25C-25E are images of a slantedpre-warped target pattern at various camera rotation angles.

DETAILED DESCRIPTION

In accordance with various aspects and embodiments of the inventiondescribed herein, improved methods and assemblies are provided forcalibration, alignment, color/white balance adjustment, and diagnostictesting of imaging devices. Such methods and assemblies can beparticularly advantageous when used with respect to an endoscopicimaging system prior to use.

Imaging Systems

Although embodiments are described with reference to applications in aminimally invasive surgical system employing an image capturing devicein the form of an endoscope, it is to be understood that the field ofthe invention is not necessarily limited to these applications. Forexample, embodiments can be used to calibrate imaging devices ingeneral.

Referring to the drawings, and with specific reference to FIG. 1, animaging system, in accordance with embodiments, is generally indicatedby reference numeral 10. System 10 includes a stereo imaging device inthe form of a stereo endoscope 12, for example. The system 10 furtherincludes two Charge Coupled Devices (CCDs) 14 and 16, optical lenses 18and 20, and read means 22, 24 for reading the CCDs and convertinginformation read on the CCDs into a digital format. The read means 22,24 is typically an appropriate electronically driven system such as aCamera Control Unit (CCU) that transforms optical information read fromthe CCDs 14, 16 into digital format. The CCD and CCU arrangements can beof the type available from Panasonic® under the part nos.:GP-US522/GP-US532 3CCD color CCU. Accordingly, an electronic processor(not shown) is typically in operative communication with the read means22, 24 as indicated by lines 26, 28. Optical lenses 30, 32 are disposedat a distal viewing end of endoscope 12. Images are passed through thelenses 30, 32, are passed along optical paths indicated by arrows 34, 36in endoscope 12, are magnified through lenses 18, 20, and are thenprojected onto optically sensitive surfaces of the CCDs 14, 16, asindicated by arrows 38, 40. Although imaging system 10 is shown anddescribed, it will be appreciated by one skilled in the art that variousalternative imaging systems can alternatively be used.

Calibration Assemblies

FIG. 2 diagrammatically illustrates a calibration assembly 50 that canbe used to generate calibration/alignment data 52, color/white balancedata 54, and/or diagnostics data 56. Calibration assembly 50 includes animaging system 58, an interface 60, a target 62, and an electronic dataprocessor 64. Imaging system 58 can include any number of devices, suchas a stereo endoscope, that can be used to capture an image and outputimage data or an image signal in response thereto. Interface 60 providesa means by which to constrain the imaging system 58 relative to thetarget 62. Interface 60 can include, for example, a lock or othermechanical constraint to prevent relative motion between the imagingsystem and the interface. Interface 60 can be coupled with target 62 sothat the target is posed (positioned and oriented) relative to theconstrained imaging system 58 so that the target is within the field ofview of the imaging system. In some embodiments that will be describedin more detail below, the interface 60 and target 62 are coupled so asto form a calibration fixture. Processor 64 is coupled with imagingsystem 58 so as to receive the image data/signal 66. Processor 64 usesthe image data/signal 66 to generate calibration/alignment data 52,color/white balance data 54, and/or diagnostic data 56.Calibration/alignment data 52 can include unique correspondences betweenextracted image features and features of target 62.

Calibration assembly 50 can include additional optional components. Forexample, the interface 60 and the target 62 can be coupled by way of amotorized mechanism. The motorized mechanism can be driven so as to besynchronized with the capture of images, such as by rotating the targetrelative to the interface between image captures and stopping during animage capture. The motorized mechanism can also be synchronized with thefocus of the imaging system 58. It has been observed that cameracalibration parameters can differ when the focus is changed. As such, animaging system may need to be calibrated at multiple focus settings. Inthis circumstance, even more images will need to be taken and amotorized mechanism may be of even greater benefit in reducing theworkload on a human operator. Interpolation can be used to determinecalibration parameters in between the calibrated focuses.

Calibration assembly 50 can include an optional user interface 68. Userinterface 68 can be used to guide a human operator during the imagecapture process. The user interface can include a communication device,such as a display or speaker, that can be used to guide the operator toposition the target relative to the imaging system. For example, userinterface 68 can be used to guide the operator to rotate the targetrelative to the imaging device by a certain angle, such as by showingthe current orientation and the desired orientation. The user interface68 can inform the operator to keep the image device fixed relative tothe target during image capture so that no motion blur occurs, which canbe especially important during modular transfer function (MTF)estimation where motion blur may not be discernible from the blur due tothe optical system.

Calibration Fixtures

FIG. 3A is a cross-sectional view of a calibration fixture 70 that canbe used to provide constraining engagement with an imaging device, suchas a zero-degree endoscope 72, in accordance with an embodiment.Calibration fixture 70 includes a receptacle 74 shaped to interface withan imaging device. Although receptacle 74 can be cylindrical, othernon-cylindrical configurations can be used. Calibration fixture 70includes a target surface 76 on which target features are located.Target surface 76 can directly incorporate the target features, or atarget containing target features can be mounted on target surface 76.Target surface 76 shown is a single planar surface, but can be anynumber of planar or non-planar surfaces. Calibration fixture 70 isconfigured so that the target features are posed relative to one or morefields of view 78, 80 of an endoscope when the endoscope is constrainedby the calibration fixture 70. The target surface 76 can be orientedrelative to the as-constrained viewing direction of the endoscope sothat a target surface normal direction 82 is at an angle 84 with the asconstrained viewing direction 86.

Calibration fixture 70 is configured so that target surface 76 can beimaged from multiple imaging directions. In use, an endoscope isinserted into the receptacle 74, which can be configured so as toconstrain the endoscope while allowing the endoscope to rotate relativeto the receptacle 74. One or more images of the target features canthereby be taken at one or more relative angular orientations betweenthe endoscope and the target surface 76.

When calibration fixture 70 is used with a zero-degree endoscope 72,rotating the endoscope relative to the receptacle 74 does not change thebasic alignment between the field of views 78, 80 and the receptacle 74.However, as shown in FIG. 3B, when the calibration fixture 70 is usedwith an angled endoscope, such as thirty-degree endoscope 88 shown,rotating the endoscope relative to the receptacle 74 does change thebasic alignment between the field of views 78, 80 and the receptacle 74,which may result in different portions of target surface 76 being withinthe fields of view for any particular relative rotation. Although suchpartial target imaging may not be optimal, it may still providesufficient image data for the generation of calibration/alignment data,color balance data, and/or diagnostics data.

FIG. 4A is a cross-sectional view of a calibration fixture 90 that canbe used with a thirty-degree endoscope 92, in accordance with anotherembodiment. Calibration fixture 90 can include an interface 94 shaped tointerface with thirty-degree endoscope 92 so as to constrain thethirty-degree endoscope 92 in one or more poses (position andorientation) relative to the target 96. For example, the interface canbe a basically cone-shaped receptacle that can be configured to providea thirty-degree endoscope 92 with a more balanced view of the target ascompared to calibration fixture 70 of FIG. 3B. For example, theinterface 94 can be configured to align the fields of view 98, 100 alonga reference direction 102. Various configurations for the interface 94can be used so as to constrain the thirty-degree endoscope 92 in one ormore poses relative to the target so that the endoscope can capture oneor more images that can be processed to generate calibration/alignmentdata, color/white balance data, and/or diagnostics data.

FIG. 4B is a cross-sectional view of a calibration fixture 110 that canbe used to with either a zero-degree endoscope 112 or a thirty degreeendoscope 114. Calibration fixture 110 combines aspects of thecalibration fixture 70 of FIG. 3A and the calibration fixture 90 of FIG.4A. Calibration fixture 110 includes an upper interface part 116 that iscoupled with a lower part 118 by way of a coupling element 120. However,the basic functionality of calibration fixture 110 can be provide in avariety of ways. For example, calibration fixture 110 can be configuredas an integral unit having multiple integrally formed receptacles, orcalibration fixture 110 can be fabricated from a number ofsubassemblies.

FIG. 5 is a cross-sectional view of a calibration fixture 130 that canbe used to provide constraining engagement with an endoscope, inaccordance with another embodiment. Calibration fixture 130 isconfigured for use with either a zero degree endoscope 132 or athirty-degree endoscope 134. With the zero-degree endoscope 132, fieldsof view 136, 138 are aligned with the zero-degree endoscope 132. Withthe thirty-degree endoscope 134, fields of view 136, 138 are misalignedwith the thirty-degree endoscope 134 by approximately thirty degrees.Accordingly, calibration fixture 130 includes a thirty-degree endoscopereceptacle 140 and a zero-degree endoscope receptacle 142, either ofwhich can be used as appropriate for the endoscope being used to captureimages of target 144 disposed on target surface 146. The thirty-degreeendoscope receptacle 140 can include features that can be used to placethe thirty-degree endoscope 134 in an angular orientation so that thefields of view 136, 138 are directed towards the target 144. Forexample, the thirty-degree endoscope receptacle 140 and thethirty-degree endoscope 134 can include reference markings (not shown)that can be aligned, or can include complementary mating features (notshown) that produce the alignment. However, exact alignment is notcritical. Calibration fixture 130 includes an upper portion 148 and alower portion 150. The lower portion 150 and the upper portion 148 arecoupled for relative rotation.

Calibration fixture 130 is configured so that target 144 can be imagedfrom multiple imaging directions. When used with a zero-degree endoscope132, the zero-degree endoscope 132 can be rotated relative to thezero-degree endoscope receptacle 142, or the lower portion 150 can berotated relative to the upper portion 148. When used with athirty-degree endoscope 134, the lower portion 150 can be rotatedrelative to the upper portion 148. A lock can be provided to preventrelative motion between the endoscope and the receptacle, which may bemore convenient for single hand operation.

FIGS. 6A, 6B, and 6C are cross-sectional views of an articulatedcalibration fixture 160. Calibration fixture 160 can be used with eithera zero-degree endoscope 162, or a thirty-degree endoscope 164.Calibration fixture 160 includes an upper portion 166 having azero-degree endoscope receptacle 168 and a thirty-degree endoscopereceptacle 170. Fixture 160 is configured so that the angle between theviewing direction 172 and the target 174 is varied as the target 174 isbeing rotated relative to the upper portion 166. The ability to controlthe slant angle of the target 174 relative to the viewing direction 172may be advantageous where different slant angles are desired. Fixture160 includes a lower portion 176 that is coupled with the upper portion166 so as to allow relative rotation about the viewing direction 172.The lower portion 176 includes an articulated target 174 that is coupledwith the lower portion at a pivot point 178. The slant angle of thetarget plane is controlled by a first rod 180 and a second rod 182,which interface with a cam surface 184 of the upper portion 166 so as toprovide varying slant angles in response to relative rotation betweenthe upper portion 166 and the lower portion 176.

FIG. 6A illustrates the relative positions of the upper portion 166, thelower portion 176, and the target 174 when the upper portion 166 is at azero-degree reference position relative to the lower portion 176. At thezero-degree reference position, the first rod 180 has been displaceddownward through contact with the cam surface 184, and the second rod182 has been displaced upwards as provided for by the cam surface 184.The positions of the first rod 180 and the second rod 182 result in thetarget slant angle shown.

FIG. 6B illustrates the relative positions of the upper portion 166, thelower portion 176, and the target 174 when the upper portion 166 is at a90-degree reference position relative to the lower portion 176 (i.e.,the upper portion 166 has been rotated 90 degrees from its position ofFIG. 6A). At the 90-degree reference position, the first rod 180 and thesecond rod 182 share equal vertical locations as provide for by the camsurface 184, which results in a horizontal target 174.

FIG. 6C illustrates the relative positions of the upper portion 166, thelower portion 176, and the target 174 when the upper portion 166 is at a180-degree reference position relative to the lower portion 176 (i.e.,the upper portion 166 has been rotated 180 degrees from its position ofFIG. 6A). At the 180-degree reference position, the second rod 182 hasbeen displaced downward through contact with the cam surface 184, andthe first rod 180 has been displaced upwards as provided for by the camsurface 184. The positions of the first rod 180 and the second rod 184result in the target slant angle shown.

FIGS. 7A, and 7B are cross-sectional views of another articulatedcalibration fixture 185. Calibration fixture 185 is configured to changethe distance between an endoscope and the target in response to rotationbetween upper portion 186 and lower portion 187. This change in distancecan be provided by a screw mechanism 188 that couples upper portion 186with lower portion 187. Calibration fixture 185 provides the ability togather calibration data for an increased range of viewing distances.This potentially makes the estimated camera parameters accurate for anincreased range of distances.

The advantages of using a calibration fixture are summarized here. Thefixture constrains target motion from six degrees of freedom (3-D rigidtransformation; three for translation and three for rotation) to 1degree of freedom rotation. This makes control of the target much morestraightforward. The constrained motion can also guarantee thatsufficient data is obtained for a successful camera calibration byfollowing simple procedures (for example, by rotating the fixture by 360degrees). Obtaining sufficient data involves both the orientationvariation of the target and the coverage and balance of calibrationfeatures in images. The use of a calibration fixture minimizesdependence upon the user and maximizes the repeatability of thecalibration process. This is especially important with surgicalassistants who may know little about camera calibration. As anadditional advantage, because the imaging device is interfaced with thefixture through a receptacle, the geometric relationship between thehousing of the imaging device (for example, the outer cylindricalsurface of an endoscope) and the imager can be partially recovered (onlyfor rotation). This geometric relationship can be useful in image guidedintervention using a robotically controlled camera arm.

Target Designs

A variety of different target designs can be used with aspects of thecalibration assemblies described herein, such as with calibrationassembly 50 of FIG. 2. Possible target designs include an existing“checkerboard” pattern, which, as discussed above, typically requiressome manual designation of the pattern in the image. Preferably, atarget design incorporates a self-referential pattern of target featuresso that the image can be automatically processed without the need forany manual designation. More preferably still, the target designincorporates multiple discrete self-referential patterns (i.e.,markers). A self-referential pattern can include, for example, localizerfeatures and identification features. Localizer features providepositional or orientation information to determine pose/alignment of themarker and the identification features can be used to differentiatebetween different markers. Such multiple self-referential patterns canadvantageously provide for more robust calibration image processing bybeing more tolerant of partial occlusions and/or image misalignments.

The use of multiple self-referential markers provides a number ofadvantages. One advantage is that portions of the image containingdifferent markers can be separately processed, which can add a level ofrobustness to the processing of the overall image by allowing thecollection of at least some useable data where portions of the targetare not imaged or portions of the image cannot processed for somereason. Another advantage is that the target pattern may allow for theuse of a less complex calibration fixture, especially with respect tocalibration of a thirty-degree endoscope that may image differentportions of the target depending on its relative orientation to thecalibration fixture (e.g., see FIG. 3B and related discussion). Anotheradvantage is that a marker can be configured to occupy a small area,which are less affected by non-linear distortion as compared to a largerpattern.

FIG. 8 diagrammatically illustrates a 2-D self-referential marker 190having localizer features and identification features, in accordancewith an embodiment. The localizer features include four dark circles192, 194, 196, 198 and a dark bar 200. The numbers within the circlesare illustrative of position designations. The localizer features of aparticular marker can be automatically associated with resulting imagefeatures, which allows for the association of the know target relativecoordinates of the localizer features with their image coordinates.

The identification features of marker 190 include thirteen dots 202(i.e., bits). The presence or absence of a dot at a particular locationin the designated pattern is a binary indicator (e.g., if the dot existsis signifies a binary “1” for the value associated with that dot'sposition, and if the dot does not exist it signifies a binary “0” forthe value associated with that dot's position). Accordingly, in theillustrative FIG. 8 example, the values shown (“0” through “9” and “a”through “c”) are illustrative of position designations for one or morebinary numbers. The thirteen dots 202 can be segregated, with some dotsbeing used for identification data and some dots being used for errorchecking data. The presence or absence of the dots used foridentification data can be used to designate a number of unique codes(or identifications). The presence or absence of dots used for errorchecking data can be used to validate a code or identificationdetermination. In one presently preferred approach, the thirteen dotsare segregated into six dots used to carry identification information(resulting in 64 unique codes), with the remaining seven dots used forerror checking. Among the seven error checking dots, six can be set tobe the inverse of the identification dots, and the remaining dot can beused as checksum data. The rationale for this approach is to alwaysensure that there are six or seven dots that are physically present in apattern (i.e., they are set to one). This avoids an all zero (all blank)pattern as a valid code and provides alternative features that can beused to provide positional information if required. The specificidentification feature pattern illustrated (e.g., number and position ofdots), along with the illustrated manner in which identification featureinformation is coded (e.g., the use of dots), is an example of manypossible identification features (see e.g., other exemplary patternsdescribed below). For more information regarding self-referentialmarkers, see the commonly owned U.S. Pat. App. No. 61/202,084 (filedDec. 31, 2008), which is incorporated herein by reference.

A target can include multiple self-referential markers. FIGS. 9A and 9Bare two different images of a target 210 containing multipleself-referential markers, in accordance with an embodiment. The imageswere obtained using a prototype calibration fixture in accordance withFIG. 3A. The imaged target 210 includes two groups of sixteen markers,with the groups separated by a straight dark bar 212 that can be used asa slanted-edge MTF feature. The markers and the dark bar 212 are setagainst a white background that can be used for the determination of acolor/white balance adjustment for the imaging system. A portion of animage of a particular marker can be separately processed (i) so as todetermine image coordinates for one or more of the localizer features ofthe marker, and (ii) to determine the identification of the marker sothat the target relative coordinates of the marker localizer featurescan be associated with their image coordinates for use in determiningcalibration/alignment data for the imaging system. As discussed above,the positions of the marker dots in the images can also be used toformulate coordinate information for use in the generation ofcalibration/alignment data. It can be seen from FIGS. 9A and 9B, forexample, that each of the markers has a different set of dots showing inthe pre-designated pattern. It can also be seen that some of the markersshare localizer features, with some circles being used as a localizerfeature for two markers.

Image Processing

FIG. 10 is a flow diagram for a method 220 for determining MTF value(s),color/white balance parameters, and calibration/alignment parameters. Instep 222, an image of a target for a set position/orientation iscaptured by using an imaging device. The imaged target contains featureswith known target relative coordinates. Step 222 can be accomplishedusing a calibration fixture, such as one of the above describedcalibration fixtures. In step 224, the captured image (i.e., image dataand/or signal) is processed so as to determine image coordinates for thetarget features. The image coordinates are associated with the knowntarget relative coordinates by associating target features with imagefeatures. The association of target features with image features can beaccomplished in a variety of ways, such as by using one of the abovedescribed target patterns, preferably a self-referential target pattern.The target relative coordinates and associated image coordinates for theparticular captured image can be combined with any possible additionaltarget images at additional positions/orientations for use indetermining calibration/alignment parameters.

In step 226, the captured image can be processed to determine one ormore MTF values. MTF provides a measure of the imaging system'sresolution and can be used for diagnostic purposes. By comparing ameasured MTF value with a standard MTF value (i.e., an acceptable MTFvalue for the imaging system in question), a measure of thefunctionality of the imaging system can be obtained. Where insufficientresolution functionality is indicated, a status and/or failure messagecan be generated to communicate that the imaging system has degradedresolution.

An MTF value can be determined by a variety of ways known in the art.The ISO 12233 spatial frequency response evaluation method is one suchapproach, and it is based on an edge-gradient method. (For furtherdiscussion, see e.g., Peter D. Burns, “Slanted-Edge MTF for DigitalCamera and Scanner Analysis,” In Proc. IS&T 2000 PICS Conference, pg.135-138, 2000.) An edge-gradient method involves the imaging of an edgefeature and processing the image of the edge feature. A key stepprocessing the image of the edge feature is the determination of thelocation and direction of the edge feature, because this act has adirect effect on the computed spatial frequency response (SFR).Advantageously, the known location and orientation of the slanted-edgeMTF feature 212 in the above described target patterns of FIGS. 9A and9B relative to the self-referential markers can be used in thisdetermination.

MTF values can be computed for a variety of directions and a variety ofpositions for each captured image. As such, a collection of MTF valuescan be computed so as to provide sufficient data regarding the health ofthe imaging system.

In step 228, the captured image can be processed to determinecolor/white balance parameters. As will be discussed in more detailbelow, the target patterns of FIGS. 9A and 9B advantageously include awhite background, which enables the determination of white balanceparameters.

If one or more additional images of the target are required for one ormore additional set positions/orientations, steps 222, 224, 226, and 228can be repeated. Steps 222, 224, 226, and 228 can be individuallyincluded or omitted as required. For example, it may not be necessary torepeat step 228 for additional positions where the white balanceparameters were sufficiently determined by using a previousposition/orientation.

In step 230, the resulting collection of associated target coordinatesand image coordinate can be used to determine calibration/alignmentparameters. In the case of a stereoscopic imaging device, calibrationparameters for the two imaging paths can be use to determine alignmentparameters. To determine alignment parameters, a virtual 3-D point canbe placed in the middle of the camera view volumes with its depth beingat a desired distance. The 3-D point is then projected into image pointsby the camera models for the left and right eyes. The difference betweenthe two image points in the image coordinates are the alignmentparameters. If necessary (e.g., due to optical assembly inaccuracy,difference in left and right eye optics), the rotation, scale, andperspective effect can also be compensated for to make for a betterviewing experience from the stereo viewer, if the camera parameters areknown.

FIG. 11 is a flow diagram for a method 240 for processing an image thatincludes two-dimensional self-referential markers of FIGS. 8, 9A, and 9Bso as to associate target coordinates with image coordinates. Ingeneral, the processing of images of such markers can use the systemsand methods described in commonly owned U.S. Pat. App. No. 61/202,084(filed Dec. 31, 2008), which was incorporated by reference above. Instep 242, an image is processed to detect dark circle localizerfeatures. In step 244, localizers 0 and 1 are identified by searchingfor two dark circles (designated in FIG. 8 as “0” and “1”) within aminimum and maximum distance and that have a bar (e.g., bar 200)generally between them that is aligned with a line connecting the twocircles. By identifying the side of the line that the bar is on, apartial orientation of the pattern can be determined (i.e., about a linein the image). In step 246, localizers 3 and 4 are identified bysearching for two dark circles (designated in FIG. 8 as “3” and “4”)within a search area relative to the identified localizers 0 and 1. Whena calibration fixture is used that constrains the target imagingdirection to a predetermined direction for any particular angularposition of the target (i.e., calibration fixtures as in FIGS. 3A, 4A,4B, 5, 6A, 6B, 6C, 7A, and 7B), the expected locations for localizers 3and 4 can be substantially predetermined based on the orientation of themarker as determined by localizers 0 and 1 and their associated bar. Instep 248, the marker can be identified by reading the identificationdots. Where the identification dots are used for error checking data,the error checking data can be used to validate the identification ofthe marker. In step 250, the image coordinates for marker features, suchas the image coordinates for the dark circle localizer features areassociated with their corresponding target relative coordinates.

Method 240 can include some optional steps. For example, Random SampleConsensus (RANSAC) can be used for outlier rejection. By estimating aglobal alignment transformation for each marker, one can detect theoutliers using RANSAC. (For details of RANSAC, see M. A. Fischler and R.C. Bolles, “Random sample Consensus: A paradigm for model fitting withapplications to image analysis and automated cartography,” Comm. of theACM, 24: pages 381-395, 1981, which is hereby incorporated byreference.) Additionally, the features of partially visible markers canbe used. The features (circles and dots) of partially visible markersare usually in the periphery of an image so that they may contributemore to the estimation of the distortion model than features in themiddle of the image. By using a first iteration of calibrationparameters, the image locations of the features which are not used inthe first iteration are known. A conservative strategy (small distancethreshold) can be used to collect such features from the images. All thefeatures can therefore be used for a second calibration iteration.

FIGS. 12A, 12B, 12C, 12D, and 12E illustrate the method of FIG. 11 for asingle marker. In FIGS. 12A and 12B, the dark circles are detected andlocalizers 0 and 1 are identified. In FIG. 12C, localizers 3 and 4 areidentified. In FIG. 12D a marker identification hypothesis is tested bychecking to see whether the image contains identification dots atexpected locations. If the marker identification hypothesis is correct,the marker is identified as illustrated in FIG. 12E. The image can alsobe processed so as to directly detect the presence or absence ofidentification dots, which can be used to directly determine theidentification of the marker.

FIG. 13A illustrates the location within an image of a calibrationtarget 260 of an exemplary color/white-balance region of interest 262that can be processed to determine color/white-balance parameters. Witha determination of the orientation and position of the target within acaptured image, a color/white-balance region of interest 262 can beselected so as to encompass a sufficient amount of the white backgroundof the target for use in determining color balance parameters, such aswhite balance parameters. As can be appreciated with reference to FIG.13A, multiple potential regions of interest exist that can be selectedfor processing to determine color balance parameters. Another approachis to extract the dark patterns and use the resulting image, whichcontains only white areas, to determine the color balance parameters(e.g., white balance parameters).

FIG. 13A also illustrates locations of exemplary MTF regions of interestthat can be processed to determine MTF values. One or more of theslanted-edge MTF regions of interest 264 disposed anywhere alongslanted-edge feature 266 can be processed to determine MTF values, whichcan provide diagnostic data for an imaging device at any selected pointalong the slanted-edge feature 266. A marker bar MTF region of interest268 can also be disposed on a marker bar 270. Accordingly, a combinationof MTF regions-of interest can be selected so as to provide diagnosticdata for multiple specific locations throughout an image. Additionally,multiple images can be processed where the slanted-edge feature 266 andmarker bars 270 have a different orientation in the image, therebyproviding additional image relative locations at which to calculate MTFvalues. FIGS. 13B and 13C illustrate a color/white balanced region-ofinterest 272 of the image of FIG. 13A and a color/white-balance control274, respectively. Region-of interest 272, for example, can be anynumber of regions-of interest that captures a background region of thetarget, such as region-of interest 262 shown in FIG. 13A, which can beselected based upon the target's position and orientation as determinedby processing one or more of the marker patterns. The region-of interest272 can be processed against the control 274 so as to determinecolor/white balance parameters. Alternatively, the dark areas can beextracted and the resulting image containing only white areas can beprocessed against the control 274 so as to determine color/white balanceparameters.

Target design variations can be used to provide slanted-edge features atadditional orientations that can be used to determine MTF values. Suchadditional slanted-edge features may reduce the number of imagesrequired to generate MTF values for vertical and horizontal image devicedirections. When determining MTF values for the vertical direction, itcan be advantageous to image slanted-edge features that are slanted by arelatively small angle (e.g., by 10 degrees) from the horizontaldirection. Likewise, when determining MTF values for the horizontaldirection, it can be advantageous to image slanted-edge features thatare slanted by a relatively small angle (e.g., by 10 degrees) from thevertical direction. In one such target design variation, the bar 200(shown in FIG. 8) can be replaced by a wedge shape having a small angle(e.g., 7 degrees), thereby providing two slanted-edge orientations perbar instead of one. Some, groups, or all of the bars on a target canalso be oriented differently, thereby providing additional orientationsfor any particular camera to target orientation. The shape of thestraight dark bar 212 (shown in FIGS. 9A and 9B) can also be modified toprovide additional slanted-edge orientations. The straight dark bar canalso be augmented with multiple instances disposed at additionalorientations (e.g. one vertical and one horizontal).

Additional Target Designs and Methods

FIGS. 14A and 14B illustrate additional calibration targets. Calibrationtarget 280 is similar in concept to calibration target 210 (shown inFIGS. 9A and 9B). Calibration target 280 uses a mixture of corners 282,dots 284, bars 286, and white balance control areas 288. Corners 282 andbars 286 can be used as localizer features. Dots 284 can be used foridentification and error-checking/correction data. A detection algorithmsimilar to method 240 (shown in FIG. 11) can be used to process an imageof calibration target 280.

Calibration target 290 provides another approach. Calibration target 290is an example of a pattern that has unique local regions (i.e., awindowed view of the whole pattern exhibits some uniqueness compared toother windowed views). Calibration target 290 is a modified version ofthe commonly used checker-board pattern. The various feature dimensions(e.g., 1, 2, 3, 4) indicate values. White regions can be used for whitebalance areas. Edges can be used for MTF calculations, as describedabove.

FIG. 15 is a flow diagram for a method 300 that can be used forprocessing an image of the calibration target of FIG. 14B. In step 302,the corners and/or edges are detected. The detected corners and/or edgescan be used to establish the neighborhood structure of the pattern. Instep 304, local areas of the pattern are located. The localization canbe done in two directions separately. A projective invariance calledcross-ratio (see chapter 2 of R. Hartley and A. Zisserman “Multiple ViewGeometry in Computer Vision,” Cambridge University Press, 2000) isdefined by four points on a line; this value does not change under anyperspective projection, and therefore it can be used to align the imagewith the calibration target. If the cross-ratio defined by the nearestfour points in not unique in the calibration target, the next fourpoints (with overlapping) can be considered/used until the four pointsused are unique in the calibration target. Methods using 2-D invariancealso exist that work on surrounding features in both directions. RANSACcan also be used for outlier rejection as described above with referenceto method 240 (shown in FIG. 11).

Target Image Pre-warping

For a rotational calibration target, the target plane needs to have anon-orthogonal angle to the camera optical axis. In order to capturesufficient data in depth, the angle can be as small as approximately 45degrees. In this case, the camera image of the target pattern has asignificant perspective effect. The perspective effect includes, but isnot restricted to, target features closer to the camera appearing to belarger than similar features that are farther from the camera, theaspect ratio of target features appearing to be changed, and parallellines on the target appearing to converge (see e.g., such effectsillustrated in the images shown in FIGS. 9A, 9B, and 13). Theperspective effect can make feature detection more difficult, since themany constraints about the feature can not be easily enforced for thedetection process. Even though feature detection processes, such as thebarcode detection described herein, can detect target pattern featuresat a variety of viewpoints, feature detection can still benefit if mostor all of the perspective effect is removed.

In the target design described herein, the rotational axis of the targetapproximately coincides with the optical axes of the cameras (there is aslight offset due to the left and right stereoscopic optical channels),and therefore the angle between the optical axes of the cameras and thetarget pattern is approximately a constant. The known angle of thetarget pattern plane with respect to the optical axis of a camera makesit possible to pre-warp the calibration target pattern by using atransformation to cancel out the perspective effect so that the patternattached to a slanted plane looks approximately like an orthographicpattern. According to camera geometry, the orthographic pattern and thewarped pattern are associated by the following equation 1:

$\begin{matrix}{\left. \begin{bmatrix}x^{\prime} \\y^{\prime} \\z^{\prime}\end{bmatrix} \right.\sim{\begin{bmatrix}d & 0 & 0 \\0 & {\cos(b)} & 0 \\0 & {\sin(b)} & d\end{bmatrix}\begin{bmatrix}x^{''} \\y^{''} \\z^{''}\end{bmatrix}}} & (1)\end{matrix}$where [x′, y′, z′] is the coordinate system attached to the orthographicpattern, [x″, y″, z″] is the coordinate system attached to the warpedpattern, d is the distance from camera center to the orthographicpattern, and b is the angle between the orthographic plane and theslanted plane, as shown in FIG. 24. It can be seen that thetransformation is not related to the focal length of the camera.

FIG. 25A is a plan view of an illustrative target pattern used forimaging and calibration as described herein. FIG. 25B is a plan (normal)view of a pre-warped version of the target pattern in FIG. 25A. Thispre-warped pattern is used for imaging, and it may be, e.g., printed andattached to the slanted target plane. Accordingly, since the anglebetween the camera axes and the target pattern plane is the same as theperspective angle of the pre-warped pattern, the imaged pre-warpedpattern generally appears as the target image shown in FIG. 25A. FIGS.25C, 25D, and 25E are illustrative images of the pre-warped targetpattern shown in FIG. 25B at three illustrative rotational angles aroundthe target's rotational axis (camera axes). As can be seen in FIGS. 25C(approximately 90-degree rotation), 25D (approximately 45-degreerotation), and 25E (approximately zero-degree rotation), these images ofthe pre-warped pattern generally look like the desired orthographicpattern shown in FIG. 25A. The majority of the perspective effect hasbeen removed.

The example patterns shown in FIGS. 25A-25E are barcode patterns. Apre-warped target pattern may be used for various other patterns (e.g.,a checker board pattern or other patterns) for the rotational target.

Imaging System Health Checks—Focus Function Metrics

A number of Focus Function derived metrics can be used to check thehealth of a imaging system. One set of Focus Function derived metricswill be described in detail below. These metrics can be appliedthroughout the lifecycle of a imaging system and can be used for avariety of purposes, such as for quality control, for detectingfailures, for detecting functional degradation that may lead to afailure, and for conducting failure analysis of imaging systemcomponents. Such metrics can be particularly beneficial in criticalapplications, such as in minimally invasive robotic surgery employing anendoscopic imaging system, where increased reliability of the imagingsystem may help to reduce the occurrences of imaging system failuresthat cause the surgeon to convert the robot-assisted minimally invasivesurgery to either open surgery or to standard laparoscopic surgery, orwhere the surgery has to be abandoned altogether. These metrics can bebuilt into an imaging system's software and performed as a routinehealth check of imaging system components. The results of these routinehealth checks can be communicated with the user and/or communicated overa communication network to a field service engineer (“FSE”) throughremote access.

A Focus Function can be produced by using an algorithm. FIG. 16Aillustrates a typical normalized Focus Function for a range of imagingsystem focal positions. The focal position that maximizes the normalizedFocus Function (i.e., the normalized Focus Function equals 1.0) is thefocal position that provides the best focus for the imaging system. Thealgorithm can be implemented with or without an encoder. Where anencoder is not used, the algorithm can use the first derivative of theFocus Function as a gain term in a proportional controller. FIG. 16Billustrates the first derivative of the Focus Function of FIG. 16A. Theuse of the first derivative as the gain term causes the gain term to goto zero at the peak of the Focus Function. The proportional controllercan implement a control loop algorithm with a “dead zone” to avoidoscillations around the peak of the Focus Function. An advantage of notusing an encoder is that there is no encoder to fail. An advantage ofusing an encoder is that the system could drive directly to a previouslyknown focal position associated with a known working distance.

FIG. 17 illustrates an algorithm 310. Algorithm 310 generates a FocusFunction for a range of imaging system focal positions by processingimages of a target of interest that is disposed at a substantiallyconstant distance from the imaging system (e.g., the target position asillustrated in FIG. 6B where the entire target surface is orientedsubstantially parallel to the plane of the objective lens of the imagingsystem). The generated Focus Function data can be used to focus theimaging system and/or can be used to generate the Focus Function derivedmetrics described below. The range of focal positions can besequentially processed, such as by starting at one end of the range(e.g., at focal position “0” of FIGS. 16A and 16B) and sequentiallyprocessing focal positions until the other end of the range is reached(e.g., focal position “800” of FIGS. 16A and 16B). Accordingly, thealgorithm 310 includes an iteration loop that starts with step 312 inwhich a focal position (“Z_(N)”) is set. An image is then captured instep 314 using the set focal position.

Where a target of interest is not disposed at a substantially constantdistance from the imaging system, a small region of interest of theimage that has a substantially constant distance from the imaging systemcan be processed. For example, in the images of FIGS. 9A and 9B, aregion of interest from both images can be selected for processing thathas a substantially constant distance from the imaging system, such as aregion of interest that contains dark line 212 and a small portion ofthe white background that surrounds dark line 212. Many such regions ofinterest can be used. For example, when a calibration fixture is usedthat results in a constant imaging direction relative to a directionnormal to the target plane (e.g., the calibration fixtures andillustrated field-of-views of FIGS. 3A, 4A, 4B, 5, 7A, and 7B), a smallregion of interest for any particular image can be selected based uponthe orientation of the target in the image so as to process a smallcommon portion of the target, which will maintain a substantiallyconstant distance from the imaging system.

In step 316, the 2-D Fast Fourier Transform (“FFT”) of the image iscalculated using known methods. As an alternative to FFT, a simplemagnitude of image gradient can be calculated using known methods. TheFFT or the magnitude of image gradient can be computed along edgefeatures only to avoid potential contributions from noisy whitebackground. In addition, the computation of the FFT or the magnitude ofimage gradient can be restricted to areas of straight edges to avoidpotential complications associated with curved features (e.g., circlesand/or dots).

In step 318, a bandpass filter is applied to the 2-D FFT data by maskingthe DC components (doesn't contain sharpness information) and the highfrequency components (noise). The cutoffs for the filter are based onthe spatial sampling rate of the image (i.e., the resolution of theimage). The mask ends up looking like an annulus.

In step 320, the remaining elements of the 2-D FFT are summed, whichproduces the Focus Function value for the set focal position (i.e.,FF(Z_(N)) is the Focus Function value for focal position Z_(N)). Steps312 through 320 can be repeated for additional focal positions in therange of focal positions to be processed. The resulting collection ofFocus Function values can be used in step 322 to focus the imagingsystem.

FIG. 18 illustrates a number of Focus-Function-derived metrics that canbe used to check the health of an imaging system, such as a stereoscopicendoscope imaging system. FIG. 18 shows Focus Function plots for astereo imaging system, which include a plot of a right eye FocusFunction 324 and a plot of a left eye Focus Function 326. The right eyeFocus Function 324 is maximized at focal position (Z_(R)) and the lefteye Focus Function 326 is maximized at focal position (Z_(L)). Ideally,Z_(L) equals Z_(R). However, there will typically be some difference(δ_(Z)) between the optics for each eye, and therefore Z_(L) willtypically be slightly different from Z_(R). If the focal positiondifference δ_(Z) is too large, then it will hinder the performance ofthe imaging system and it will be impossible to choose a focal positionthat works for both eyes. Accordingly, the focal position differenceδ_(Z) should not exceed some threshold (T_(Z)). This constraint can beused in quality control (e.g., by imposing it at the time of imagingsystem manufacture) and can be used over time as a health metric for theimaging system, as shown in equation 2 below.δ_(Z)<T_(z)  (2)

The peak value of the Focus Function represents the sharpness of theimage. Ideally, the right eye Focus Function 324 at focal position Z_(R)will equal the left eye Focus Function 326 at focal position Z_(L).However, there will typically be some difference (δ_(FF)) between thesetwo values due to differences in the optical paths. If the FocusFunction peak value difference δ_(FF) is too large, then the right andleft images will not have the same sharpness, again hindering theperformance of the imaging system. Accordingly, the difference in FocusFunction peak values δ_(FF) should not exceed some threshold (T_(FF)).This constraint can be used in quality control and can be used as anongoing health metric for the imaging system, as shown in equation 3below. If δ_(FF) exceeds T_(FF), then a warning message can be displayedto the user and/or sent over a communication network to indicate aproblem with the imaging system.δ_(FF)<T_(FF)  (3)

Prior to normalization, the peak Focus Function values can be evaluatedto determine whether they meet a minimum threshold. In one approach,prior to normalization the peak left eye Focus Function and the peakright eye Focus Function can be compared against a minimum standardFocus Function to ensure acceptable imaging resolution is beingachieved. In another approach, the Focus Function values determined canbe normalized by using a peak Focus Function value from a selectedimaging system (e.g., a system having a best resolution) and a minimumacceptable Focus Function Threshold value set accordingly.

Because the best focal position for the left eye Focus Function Z_(L)will not typically equal the best focal position for the right eye FocusFunction Z_(R), it may be necessary to select a common focal position(Z′). While the common focal position Z′ could be selected to be eitherZ_(L) or Z_(R), this would leave the eye using the unselected opticalpath out of focus. Therefore, it can be advantageous to set the commonfocal position Z′ as the midpoint between Z_(L) and Z_(R). The commonfocal position Z′ should not change over time for a given image distanceand imaging system (e.g., a particular endoscope and camera combination.Additionally, for any given imaging distance, the common focal positionZ′ can be constrained within set bounds (i.e., between a minimum commonfocal position (Z_(MIN)) and a maximum common focal position (Z_(MAX)))for all imaging systems of a particular group and/or design, such as forall endoscope/camera combinations of a particular group and/or design.This constraint can be used for manufacturing quality control and can beused over time as a quality metric on the health of the imaging system,as shown in equation 4 below.Z_(MIN)<Z′<Z_(MAX)  (4)Imaging System Health Checks—MTF and Relative MTF

Modulation Transfer Function (MTF) is a measure of contrast andresolution of an imaging system. FIG. 19 illustrates a typical MTF curvefor an imaging system. In order to ensure a minimum level of performanceof an imaging system with regard to contrast and resolution, one or moreMTF threshold values can be specified. For example, the MTF value at themaximum resolution of the imaging system (MTF(R_(MAX))) can be requiredto be greater than ten percent.

A “relative MTF” measurement can also be used to monitor the health ofan imaging system. A relative MTF measurement compares two images of anarbitrary target, thereby providing two MTF curves that can be comparedto detect any resolution differences between the two images. FIGS. 20Aand 20B show two images that were generated from a portion of a tissueimage. FIG. 20A shows the portion as taken from the original tissueimage. FIG. 20B shows the same portion, but the image was blurred usinga 4×4 Gaussian kernel. The difference in resolution between these imagesis easily distinguished with the naked eye. The images of FIGS. 20A and20B simulate two different images of a common arbitrary target taken bytwo imaging systems having different resolutions. FIGS. 20C and 20D eachshow the 2-D FFT for the images of FIGS. 20A and 20B, respectively. TheFIG. 20D 2-D FFT for the blurred image of FIG. 20B shows a loss ofhigh-frequency information as compared to the FIG. 20C 2-D FFT for theoriginal image of FIG. 20A. Also, the 2-D FFT for the blurred image hasmuch less noise components.

FIG. 21 shows a comparison between a relative MTF curve 328 calculatedfrom the 2-D FFT of FIG. 20C (i.e., the 2-D FFT for the non-blurredimage) and a relative MTF curve 330 calculated from the 2-D FFT of FIG.20D (i.e., the 2-D FFT for the blurred image). Curve 328 is disposedabove curve 330, thereby illustrating the relatively higher resolutionof the original image as compared to the blurred image. Curve 330 forthe blurred image exhibits a steep drop off 332. Such a steep drop offmay be indicative of a “resolution limit” of an imaging system used tocapture the corresponding image processed.

FIGS. 22A, 22B, 22C, 22D and 23 illustrate the ability of relative MTFcalculations to discern more subtle differences in the resolutionbetween two images. FIGS. 22A and 22C are identical to FIGS. 20A and 20C(repeated for comparison purposes). FIG. 22B shows a slightly blurredversion of the original image of FIG. 22A. The image of FIG. 22B wasblurred using a 3×3 gaussian kernel. The difference in resolutionbetween these images is not easily distinguished with the naked eye.FIG. 22D shows the 2-D FFT for the slightly blurred image. It is moredifficult to determine if the 2-D FFT for the slightly blurred image(FIG. 22D) has less high-frequency content than the 2-D FFT for theoriginal unblurred image (FIG. 22C). However, it can still be seen thatthe 2-D FFT for the original unblurred image has higher-frequencycomponents that represent noise, and that these noise components are notpresent in the 2-D FFT for the slightly blurred image.

FIG. 23 shows the resulting relative MTF curves for the originalunblurred image of FIG. 22A and the slightly blurred image of FIG. 22B.From FIG. 23, it is easier to see the resolution differences between theoriginal unblurred image (curve 334) and the slightly blurred image(curve 336). Such curves can be used to make quantitative judgmentsabout the differences in resolution between the two images and thusabout differences in resolution between the imaging systems used tocapture the two images. For example, the respective relative MTF valuesfor the two images at a particular spatial frequency can be used tocalculate a percentage contrast difference at that spatial frequency.

Relative MTF calculations can be used to be used to gain insight intothe relative frequency and contrast characteristics of two imagingsystems. The approach requires that two images be taken of the sameobject using the same field of view. The object imaged can be acalibration target, such as one of the above-described calibrationtargets. The above-described calibration targets contain sharp edgefeatures, which should have sufficient spatial frequencies to generaterelative MTF values at various resolutions. By comparing the resultingrelative MTF curves for the images and/or the resulting 2-D FFTs for theimages, relative differences between the two imaging systems can bedetermined. Similar to the above-discussed Focus Function, relative MTFcan be calculated for each image path in a stereo imaging system. Thetarget should be at a constant depth to the camera or it may bedifficult to separate the blur due to out of focus problems and otherproblems. To be used with an inclined target, a region-of interesthaving a relatively constant depth can be used.

Relative MTF calculations can be used to quantify the health of aimaging system. For example, relative MTF calculations can be used as aquality control metric during the manufacture of the imaging system, asa quality control metric during integration of the imaging system, as aroutine health check of a imaging system that can be built into systemsoftware for the imaging system, and as a failure analysis tool for useon imaging system components. The results of the relative MTFcalculations can be reported to the imaging system user and/or reportedover a communication network (e.g., to a field service engineer).

Imaging System Health Checks—Other Metrics

The amount of adjustment required to align the separate optical paths ina stereo imaging system can be used as a health check metric. Alignmentof the separate optical paths can be performed automatically using across-correlation technique. For a given target (e.g., a calibration oralignment target) and imaging system, the X shift (S_(X)) and the Yshift (S_(Y)) required to align the separate optical paths should besmaller than pre-defined thresholds (ShiftMax_X and ShiftMax_Y),respectively, as shown in equations 5 and 6, below. If an alignmentshift exceeds its threshold it may be indicative of a problem with theimaging system, such as a problem with the endoscope or the cameraassembly in an endoscopic imaging system.S_(X)<ShiftMax_X  (5)S_(Y)<ShiftMax_Y  (6)

The brightness of an image of a target can be used as a health checkmetric. The brightness of the image can depend upon the amount ofillumination delivered and the health of the image capture system. Theamount of illumination delivered can vary depending upon the health ofthe illumination system. For example, the amount of illuminationdelivered in an exemplary endoscopic imaging system can depend upon thehealth of the lamp and the health of the fiber illuminator cables. Thehealth of the image capture system can depend upon the health of imagecapture components, such as a charge-coupled device, a camera controlunit, or an optical path. By controlling the illumination system toensure that the illuminator is set at a known value (for example themaximum value) and controlling the image capture system (e.g., thecamera control unit) so that it is not compensating for the brightnessof the image of the target, the amount of illumination measured from theimage can be indicative of the health of the illumination system and/orthe health of the image capture system and compared against a thresholdlevel (T_(L)) for each optical path (e.g., illumination measured fromthe left eye image (L_(L)) and illumination measured from the right eyeimage (L_(R))), as shown in equations 7 and 8 below. When the measuredillumination level drops below the threshold level, a lamp change and/ordiagnostic maintenance can be recommended to the user. Additionally, theillumination status can be reported to the user or sent out over acommunication network.L_(R)>T_(L)  (7)L_(L)>T_(L)  (8)

It is understood that the examples and embodiments described herein arefor illustrative purposes and that various modifications or changes inlight thereof will be suggested to a person skilled in the art and areto be included within the spirit and purview of this application and thescope of the appended claims. Numerous different combinations arepossible, and such combinations are considered to be part of the presentinvention.

What is claimed is:
 1. A calibration target for use in calibrating animaging system, the target comprising: a plurality of markers with eachmarker including a plurality of localizer features and a plurality ofidentification features; and a target pattern plane having a planarsurface; wherein the plurality of localizer features have known relativepositions on the target and are used to determine an orientation foreach marker; wherein the plurality of identification features are usedto determine an identification for each marker; wherein the plurality ofmarkers is disposed on the planar surface of the target pattern plane,and wherein an optical axis of the imaging system is at a first anglewith respect to the planar surface of the target pattern plane when thecalibration target is being used to calibrate the imaging system, andwherein each of the plurality of markers is pre-warped in size andaspect ratio by using a set of trigonometric functions that use thefirst angle and a distance from the imaging system to the marker whenthe calibration target is being used to calibrate the imaging system, sothat each of the plurality of markers appears a substantially same sizeas all others of the plurality of markers when viewed by the imagingsystem at the first angle with respect to the planar surface of thetarget pattern plane when the calibration target is being used tocalibrate the imaging system.
 2. The calibration target of claim 1,wherein the identification for each marker of the plurality of markersis different than the identifications for all other markers of theplurality of markers; and wherein the plurality of identificationfeatures for a first marker of the plurality of markers includes a firstplurality of potential dot locations and a second plurality of potentialdot locations.
 3. The calibration target of claim 2, wherein theidentification for the first marker is indicated by: a presence or anabsence of a dot in each of the first plurality of potential dotlocations so as to indicate a first pattern of dots, and a presence oran absence of a dot in each of the second plurality of potential dotlocations so as to indicate a second pattern of dots which is an inverseof the first pattern of dots, wherein each potential dot location of thefirst plurality of potential dot locations is associated with acorresponding bit location in a binary number, wherein the first patternof dots indicates a first binary number by the presence of a dot in eachof the first plurality of potential dot locations indicating a binaryone for the corresponding bit location, and by the absence of a dot inthe each of the first plurality of potential dot locations indicating abinary zero for the corresponding dot location.
 4. The calibrationtarget of claim 3, wherein the plurality of identification features forfirst marker includes an additional potential dot location; and whereina checksum for the first binary number is indicated by the presence of adot in the additional potential dot location indicating a binary one andthe absence of a dot in the additional potential dot location indicatinga binary zero.
 5. The calibration target of claim 1, wherein each of theplurality of markers resides in an area of a rectangle, wherein theplurality of localizer features for a first marker of the plurality ofmarkers includes four dark circles, and wherein each of the four darkcircles resides in a corresponding corner of the rectangle in which thefirst marker resides.
 6. The calibration target of claim 5, wherein therectangle in which the first marker resides has a width of 5millimeters.
 7. The calibration target of claim 5, wherein the pluralityof markers is arranged in a matrix of rectangular-shaped markers, andwherein one side of the rectangle in which the first marker residesshares a pair of dark circles with an adjacent marker in the matrix ofrectangular-shaped markers.
 8. The calibration target of claim 5,wherein the plurality of localizer features for the first markerincludes a straight dark bar on one side of the rectangle in which thefirst marker resides, wherein the straight dark bar is set against awhite background that can be used for a determination of a color/whitebalance adjustment for the imaging system.